Illumination Reconstruction from Real-time Video for Interactive Augmented Reality

This paper presents an interactive real-time augmented reality system with realistic illumination of integrated virtual objects. A marker and a mirror sphere are placed in a scene that is captured by a video camera. Dynamic illumination (as the marker is interactively moved) is estimated from the video signal and used for realistic real-time integration of virtual objects into the video. To achieve interactive real-time performance, we have developed a hardware-accelerated approach, where self-occlusion and form factors are precalculated and stored in a low-resolution cube map for each vertex. This data is multiplied with the real-time illumination information estimated from the video to obtain diffuse per vertex illumination. Our approach works well for small virtual objects moving around in real environments and is almost completely implemented on the GPU per performance purposes. The real environment operates as a large area light source, which casts soft shadows on our diffusely lit virtual objects.